Exhaustive Testing as a Verification Technique†

For a safety-critical system, i.e., a system whose consequences of failure are very high, it is not possible to rely upon testing to provide the necessary verification. The difficulties arise mainly from the shear number of tests that are required to permit statistically meaningful conclusions to be drawn about the system. Other difficulties with testing include failure to observe erroneous output when it occurs and incorrectly defining the operational profile from which to select inputs. Goodenough and Gerhart observed that exhaustive testing of a software system amounts to a proof of the software. This is an appealing thought for safety-critical systems because establishing proofs of system properties by more traditional techniques is difficult at best and often depends on questionable assumptions such as assuming correct translation by a compiler. Unfortunately, however, if testing that yields a statistical conclusion is infeasible it would seem that exhaustive testing would be also. In general, exhaustive testing is infeasible. However, this infeasibility is a direct result of the goal of testing for overall functional correctness. In considering the general issue of how safety-critical systems might be tested, we have concluded that a different view of testing is required. The view we advocate is that testing should be used to show significant properties of safety-critical software systems rather than overall correctness. This approach to testing is analogous to the use of formal verification to demonstrate properties rather that correctness. When testing is used as a technique to establish a property, the property of interest determines in large part the number of tests required. This is the case no matter whether the goal is to establish properties in a statistical sense or in the sense of a proof using exhaustive testing. By careful definition of the property and by the application of a technique called specification limitation, we have been able to prove a number of significant properties of a large software system by exhaustive testing. In this paper we formalize the notion of specification limitation and show how it can be applied in practice. We present the details of a complex software system and associated properties that were established by applying the technique. Exhaustive Testing As A Verification Technique